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A framework for identifying schools where 

students need change — now! 



As our nation's struggling schools attract greater attention and investments, education 
leaders need to understand trends in school performance over time. 

Not all low-performing schools are the same. Some remain stuck year after year, while 
others that started as low performers are among the fastest improvers in their state. 

Educators and policymakers can use the Stuck Schools framework to identify schools 
worthy of study as well as those in need of intervention. 



£ The Education Trust 

IXJ 





The Stuck Schools analysis offers a simple, 
transparent way of gauging patterns of school 
performance and improvement. It shines a 
light on a relatively small number of schools 
that started out low performing and have 
made little or no gains in subsequent years — 
schools that clearly need state or district 
attention. Tracking actual proficiency rates 
and their improvement over time can help 
decision makers focus scarce resources on a 
limited number of struggling schools and the 
students they serve. 
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Stuck Schools 

A framework for identifying schools where students need change — now! 

BY NATASHA USHOMIRSKY AND DARIA HALL 



Consider two middle schools in Maryland. At both, low-income 
kids and kids of color constitute the majority of the student popu- 
lation. In 2005, about one-third of the students at each school 
tested proficient in reading on the Maryland State Assessment. 

By 2009, however, 64 percent of students at one school reached 
proficiency, while proficiency rates remained below 40 percent at 
the other. 

To be sure, a 64 percent proficiency rate is not nearly good 
enough. Yet gains of eight percentage points per year (some of 
the highest in Maryland) suggest that something positive may 
be happening at this school. In contrast, little appears to have 
changed at its counterpart, where annual gains averaged less 
than a percentage point for five consecutive years. Both schools 
may require additional help and resources, but their needs prob- 
ably differ in type and intensity. 

lot of claims and counterclaims are being made 
about how much America's schools — especially 
our low-performing schools — are improving. 
Some argue our country has made literally no 
progress on this issue, either because we don't know how 
to improve schools or because we don't care enough to do 
it to scale. Others say things are getting better across the 
board, pointing to what they see as unprecedented gains in 
initially low-performing schools over the past few years. 

In recent months, the federal government has put bil- 
lions of dollars on the table with a demand for real action 
in turning around our country's lowest performing schools. 
At the same time, federal and state leaders are consider- 
ing future directions for education policy. In this context, 
understanding recent patterns of school improvement is 
particularly important. 
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To help inform that conversation, we have explored 
data on reading and mathematics performance over time 
in ten states. In each state, we identified elementary and 
middle schools where math or reading-proficiency rates a 
few years ago ranked among the lowest. Then we examined 
the extent to which these initially low-performing schools 
raised or failed to raise students' reading and math perfor- 
mance during subsequent years. 

In virtually every state, the story parallels that of the two 
Maryland middle schools: Some schools are improving; 
others are stuck. 

The data convey some important messages. On the 
one hand, the notion that no one knows how to improve 



The Stuck Schools Series 

The four papers in the Stuck Schools Series provide educa- 
tors, policymakers, and the public with a framework for 
using data to identify schools and districts that are making 
academic progress or that desperately need help. 

• This paper, the first in the series, looks at trends in 
overall performance and improvement overtime. It 
examines data from two states to determine what 
performance looked like several years ago, the extent 
of annual gains at high-improving and low-improving 
schools, and how many of these lowest performing 
schools remained stuck, made extraordinary gains, 
or fell somewhere in between. 

• The second paper will look beyond overall test scores 
and consider the performance of subgroups of stu- 
dents. 

• The third will explore what the data say about perfor- 
mance and improvement trends across districts. For 
example: Are some districts more successful than 
others in moving low-performing schools? 

• Finally, a fourth paper will address the public-policy 
implications of these analyses. 
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low-performing schools doesn't hold up. Degrees of prog- 
ress differ from state to state, but a substantial number of 
schools in nearly every state are making great strides. Of 
course, rising test scores alone are insufficient cause for 
celebration: It's important to identify the instructional prac- 
tices that are spurring these results. But the data indicate 
that some schools have the capacity and knowledge to drive 
success. It's essential to find out what these schools are 
doing right and replicate the practices that lead to meaning- 
ful, sustained gains for all students. 

But let's not kid ourselves. Academic performance and 
improvement still fall short of where they should be, and 
progress differs drastically among the states. For example, 
on average, reading and math proficiency rates in Maryland 
have improved substantially in recent years, yet in other 
states — Indiana, for example — average performance has 
remained flat. 

What's more, every state has a group of schools that 
started out low performing and proceeded to show little 
improvement — or all too often, lose ground. These schools 
are stuck — trapped at the bottom and unable or unwill- 
ing to improve — and tens of thousands of students are 
stranded inside their classrooms. Identifying these schools 
and either helping them improve or forcing them to close, 
with students removed to higher performing schools, is an 
economic and moral imperative. 

With the focus of the American Recovery and Reinvest- 
ment Act (ARRA) on struggling schools and reauthorization 
of the Elementary and Secondary Education Act (ESEA) on 
the horizon, now is the time to have an honest conversa- 
tion about what we know — and don't know — about school- 



Clarifying Our Terms 

The terms performance, proficiency rates (or percent profi- 
cient), passing rates (or percent passing), and test results all 

refer to the percentage of students who earned proficient 
scores or higher in reading or mathematics on a state test in 
a given year. 

Similarly, the terms gains and improvement both mean the 
increase or decrease in the percentage of students proficient 
in reading or math from year to year. 

High-performing, average, and low-performing schools 
referto schools that in the baseline ranked in the top quartile, 
middle 50 percent, and bottom quartile of performance, 
respectively. High-improving, average, and low-improving 
schools refer to schools whose gains from 2005-09 ranked in 
the top quartile, middle 50 percent, and bottom quartile of 
improvement. 



The notion that no one 
knows how to improve low- 
performing schools doesn't 
hold up. Degrees of progress 
differ from state to state, but a 
substantial number of schools 
in nearly every state are 
making great strides. 

level trends. To spur this conversation and to help state 
and local leaders think about how to look at their own 
data, The Education Trust will publish a series of papers 
that analyze school performance and improvement. 

In this paper, the first in the series, we look at trends 
in overall performance and improvement over time. We 
examine data from two of the ten states we looked at to 
explore the following questions: What did performance 
look like several years ago? Flow big were the annual gains 
at high-improving schools? How about at low-improving 
ones? Among the lowest performing schools, how many 
remained stuck, how many made extraordinary gains, and 
how many fell somewhere in between? 

FINDING THE STUCK SCHOOLS 

What follows is a brief overview of how we analyzed data 
from Maryland and Indiana, our example states; Appendix 
I provides additional detail, including all data sources. 

Using publicly available data, we calculated school- 
level aggregate proficiency rates in reading and math for 
five consecutive years (2004-08 in Indiana and 2005-09 
in Maryland). 1 After eliminating schools that were miss- 
ing data for one or more years, we calculated the baseline 
proficiency rate and average annual improvement rate for 
each school in each subject. 2 We then ranked schools by 
their baseline proficiency rates, identifying schools that 
were high performing (top quartile), average (middle 50 
percent), and low performing (bottom quartile) in reading 
and math, respectively. This gave us a picture of the range 
of performance across the state. 

Next, we ranked schools by their improvement rates 
(ignoring their performance for the moment) and identi- 
fied schools that were high improving, average, and low 
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improving in each subject — that is, in the top quartile, 
middle 50 percent, or bottom quartile of improvement. 
This helped us better understand the kinds of improve- 
ment trajectories seen in schools. 

Finally, we combined these two analyses to find out 
how schools that were low performing in a given subject 
in the baseline ranked in terms of improvement in that 
subject. This allowed us to identify two sets of schools — 
those that started out low performing but were among the 
top gainers in the state (though often still short of their 
potential) and those that started out low performing and 
have improved more slowly than 75 percent of schools in 
the state. This latter group comprises the stuck schools. 

WHAT THIS ANALYSIS DOES— AND DOESN'T DO 

This analysis offers a simple, transparent way of gauging 
the patterns of school performance and improvement in 
a state. It shines a light on a relatively small number of 
schools that started out low performing and have made 
little to no gains in subsequent years — schools that clearly 
need state or district attention. In doing so, this analysis 
demonstrates that tracking a combination of actual pro- 
ficiency rates and their improvement over time can help 
educators and policymakers focus the scarce resources of a 
district or state on a limited number of struggling schools 
and the students they serve. 

Of course, this analysis alone cannot determine which 
schools to close or target for turnaround. For one, it 
does not take subgroup performance into account, and 
schoolwide averages can mask huge gaps between groups 
of students. (The second paper in the series examines the 
issue of subgroup achievement.) The analysis also is based 
entirely on state assessment results, which come with their 
own limitations. In addition, proficiency rates of small 
schools can fluctuate a great deal from year to year. 

Moreover, this analysis is meant to bolster — not 
confuse — the accountability and school-identification 
policies already in place through the ESEA and the ARRA. 
Our approach to looking at achievement data com- 
pares schools with one another, rather than against a set 
standard. The Adequate Yearly Progress (AYP) provisions 
of the ESEA, on the other hand, use a target benchmark- 
proficiency rate and evaluate all subgroups of students 
against that common standard to determine each school's 
status. Under AYP, a school's status is based on its own 
performance against a set target, whereas our analysis 
bases a school's status on its performance relative to other 



schools in the state. Because of these differences and other 
complexities of AYP, our analysis and the AYP formula 
identify somewhat different groups of schools as needing 
support, though considerable overlap exists between the 
two approaches. 

While similar in intent to the turnaround provisions 
of Race to the Top and School Improvement Grants, this 
analysis will not necessarily yield the 5 percent of schools 
the Obama Administration hopes to address with these 
programs. In some states, our approach will identify far 
more schools; in others, far fewer. Regardless, state officials 
can use this framework to help identify schools in need of 
intervention and those that have lessons to offer. 

PATTERNS IN MARYLAND AND INDIANA 

This paper examines elementary and middle school per- 
formance over time in Maryland and Indiana. We selected 
these states because they illustrate contrasting patterns of 
performance and improvement — patterns that, to various 
degrees, are evident in multiple states and subject areas. 

A caveat: Because of differences in content and per- 
formance standards, as well as the different years of data 
included in the analysis, we cannot compare one state's 
achievement and improvement results with another's. We 
are not seeking to determine that one state is doing better 
or worse than another but rather to provide educators and 
policymakers with examples of what to seek — and watch 
out for — in their own data. 

That said, at least according to a recent study by the 
Nation Center for Education Statistics (NCES), 3 read- 
ing performance standards in Maryland and Indiana are 
not drastically different in terms of difficulty. And while 
according to NCES, Indiana's fourth-grade math perfor- 
mance standards are substantially higher than Maryland's, 
its eighth-grade standards are actually somewhat lower. 
Our discussion focuses primarily on results in reading; 
mathematics charts and tables for both states are available 
in Appendix II. 

MARYLAND: MANY GAIN, SOME STAGNATE 4 

Our analysis for Maryland included 2005-09 assess- 
ment results for 1,066 schools serving any combination 
of grades 3-8. It excluded any schools that served one or 
more high school grades (9-12), as well as alternative 
schools and schools where 50 percent or more students 
received special education services. 5 
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Figure 1: Baseline (2005-07) Reading Proficiency Rates of Figure 3: Demographics of Maryland's High-Performing, Average, 

Maryland's High-Performing, Average, and Low-Performing and Low-Performing Schools: Reading 
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Figure 2: 2005-09 Improvement Rates at Maryland's High- Figure 4: Demographics of Maryland's High-Improving, Average, 

Improving, Average, and Low-Improving Schools: Reading and Low-Improving Schools: Reading 
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Baseline Reading Performance. On average, during 
the 2005-07 baseline period, 77 percent of Maryland's 
third-grade to eighth-grade students passed the Maryland 
State Assessment (MSA) in reading. At high-performing 
schools (schools whose proficiency rate surpassed the 75th 
percentile school's), an average of 92 percent of students 
were proficient in reading. The average profiency rate at 
the middle 50 percent of schools was 79 percent. At low- 
performing schools (schools whose proficiency rate was 
lower than the 25th percentile school's) only 58 percent of 
students, on average, passed the MSA (see Figure 1). 

2005-09 Improvement. From 2005-09 in Maryland, the 
percentage of elementary and middle school students 
who scored proficient or above in reading on the MSA 



increased on average by about 2.8 percentage points per 
year. The highest improving 25 percent of schools gained 
an average of 5.6 percentage points per year, or more than 
20 percentage points over this period. Low-improving 
schools (schools in the bottom quartile of improvement), 
on the other hand, gained only about one-half a percent- 
age point per year on average — or about two percentage 
points over the entire five years (see Figure 2). 

Demographics of Schools Differing in Performance and 
Improvement. Figure 3 shows the demographics of high- 
performing, average, and low-performing schools. Dis- 
hearteningly, but not unexpectedly, schools in the bottom 
quartile of performance have the highest percentages of 
minority and low-income students. 
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Figure 5: 2005-09 Reading Performance of Maryland High- Figure 6: Number of Low-Performing Schools That Are High- 

Performing, Average, and Low-Performing Schools Improving, Average, and Low-Improving: Reading 



100 




20 - 

0 

2005 2006 2007 2008 2009 



^High Performing 0 Average Performing ■ Low Performing 



Low Improving, 
19(7%) 




In contrast, the demographics of high-improving, aver- 
age, and low-improving schools (see Figure 4) show the 
opposite pattern: High-improving schools have the highest 
percentages of low-income and minority students. The fact 
that schools serving low-income students and students 
of color are more likely to be high-improvers than other 
schools offers an encouraging sign of progress toward the 
goal of closing achievement gaps. 

Where Schools at Different Performance Levels Started 
and Ended Up. Figure 5 shows how schools that started 
out high performing, average, and low performing fared 
on the MSA during 2005-09. On average, the news is quite 
good: Schools at each level of performance generally made 
progress, with low-performing schools making the biggest 
gains. 

While this may seem obvious — schools that start the 
lowest have the most room to improve — not all schools 
that begin low make gains. As Figure 6 shows, about 64 
percent of Maryland's low-performing elementary and 
middle schools rank as high improving (meaning, they are 
improving faster than three-fourths of the schools in the 
state). Another 77 schools, or 29 percent of the total, show 
up as average-improving. But 19 schools, 7 percent of the 
low performers, are also among the slowest improving in 
the state. During 2005-09, these 19 schools gained less 
than half a percentage point on average, and some have 
consistently declined in performance. 

Figure 7 highlights the divergent trends among the 267 
schools that started out as low performing. The highest 



Figure 7: 2005-09 Reading Proficiency Rates of Maryland's 
Low-Performing Schools Showing High, Average, and Low 
Improvement 
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improving among them attained a 76 percent proficiency 
rate, on average, by 2009 — almost reaching the baseline 
state average. But the lowest improving stagnated at just 
about 60 percent proficiency. In other words, while some 
schools that started low-performing have made substantial 
progress, others have made no gains at all. And notably, 
among this low-improving group, some schools began and 
ended in the 30 percent to 40 percent proficient range. 

When Gains Aren't Enough: Chronically Low-Performing 
Schools. Our "stuck schools" definition identifies schools 
that started out performing poorly and improved more 
slowly than three-fourths of all schools in the state over a 
period of five years. This definition includes many schools 
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that showed limited to nonexistent improvement, but it 
fails to capture schools that remain among the lowest per- 
forming in the state, despite making slightly higher gains. 
When a school continues to deliver worse results than the 
vast majority of schools in the state year after year, its status 
at the bottom of the performance continuum should spark 
notice, even if it is making some gains. Such schools are 
chronically low performing and need attention, despite the 
fact that they are not "stuck." 

To identify Maryland's chronically low-performing 
schools, we looked for schools whose 2007, 2008, and 
2009 proficiency rates stayed consistently below the per- 
formance of the fifth-percentile school in the baseline. (See 
the sidebar, "Using Baseline Performance as a Ruler.") In 
2007, 40 schools fell below this benchmark for reading. Of 
these, seven schools performed below this benchmark in 
2008 and 2009 as well (Figure 8). 

Tallying Stuck and Chronically Low-Performing Schools. 

By looking at reading performance over time, we have 
identified 19 Maryland schools that are stuck and seven 
that are chronically low performing. Figure 9 shows some 
overlap between these two groups of schools, however. In 

Figure 8: Maryland's Chronically Low-Performing Schools: Reading 

Number of Schools 



Exited bottom 5 percent 31 



Exited bottom 5 percent but declined again 2 



In bottom 5 percent for all three years 



Using Baseline Performance as a Ruler 

When identifying chronically low-performing schools, we 
looked for schools whose performance in the last three 
years of the time period analyzed fell consistently below that 
of the fifth-percentile school — not in each of those years but 
in the baseline. 

We used baseline performance as our "ruler" for several 
reasons. First, as proficiency rates in a state improve, the 
passing rate that schools need to reach to emerge from the 
bottom 5 percent tends to rise. Setting the bottom 5 percent 
bar based on a single time period allowed us to compare 
each school to a consistent target, rather than a moving one. 
Using baseline data ensured that we identified schools that 
were lagging so far behind that they required state or district 
attention, even though they may have been making some 
gains. 



Figure 9: Overlap Between Maryland Schools That Are Stuck 
and Chronically Low Performing in Reading 




Low Performing 



Figure 10: Overlap Between Maryland Schools Identified as 
Chronically Low Performing or Stuck in Reading and Math 




Identified in both Reading 
and Math 



fact, four of the seven chronically low-performing schools 
are among the lowest improving in the state and are 
captured in the stuck-schools analysis. The remaining three 
schools are making higher, though insufficient, gains. In 
total, 22 elementary and middle schools in Maryland are 
stuck, chronically low performing, or both. 

In addition, 2005-09 MSA results indicate that 31 
schools are stuck or chronically low performing in math 
(see Appendix II). In total, 44 schools, or about 4 percent 
of the elementary and middle schools we studied, are stuck 
or chronically low performing in math, reading, or both. 
Nine are stuck or chronically low performing in both math 
and reading, 13 in reading only, and 22 in math only (see 
Figure 10). 
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Although overall test results tell only part of the story, 
looking at the data in a variety of ways — finding schools 
that are both stuck and chronically low performing or that 
are identified as such in both subjects — can help focus 
state and local attention even more acutely. Who would 
argue that students in a school that's consistently in the 
bottom 5 percent of performance and improving slower 
than three-quarters of schools in the state in both reading 
and math don't deserve something better? 

Trends in Maryland's Data. Maryland's data display some 
encouraging trends in school performance. On average, 
reading and math proficiency rates are improving from 
year to year, and schools that started out low performing 
generally are making bigger gains than those that started 
out as high performing. 

Among schools that started as low performing, how- 
ever, a small number are not improving or are improving 
far too slowly. What's more, our analysis suggests that 
certain districts are less effective than others in advancing 
their low-performing schools. The issue of district capacity 
has received insufficient attention in conversations about 
school improvement. (The third paper in this series will 
explore district patterns in more detail.) 

INDIANA: STAGNATING STATEWIDE 

The Indiana analysis included 2004-08 assessment results 
for 1,477 schools. As in Maryland, alternative and special 
education schools were excluded from the dataset. For 
schools serving some combination of grades 3-8 that also 
served one or more high school grades, we used test results 
for third to eighth grade only. 

Baseline Reading Performance. On average, in 2004-06 
(the baseline) 72 percent of Indiana's third-grade to 
eighth-grade students passed the reading portion of the 
Indiana Statewide Testing for Educational Progress-Plus 
(ISTEP+). The average proficiency rate at high-performing 
schools was 85 percent; at low-performing schools the pro- 
ficiency rate averaged about 57 percent (see Figure 11). 

2004-08 Improvement. Unlike in Maryland, the average rate 
of reading proficiency in Indiana remained virtually flat 
from 2004 to 2008. On average, schools in the top quartile 
of improvement gained only 2.2 percentage points per 
year, while schools in the bottom quartile of improvement 
lost 1.9 percentage points annually. This means that pro- 
ficiency rates among low "improvers'' declined by nearly 
eight percentage points over this period (see Figure 12). 



Figure 11: Baseline (2004-06) Reading Proficiency Rates of 
Indiana's High-Performing, Average, and Low-Performing Schools 
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Figure 12: 2004-08 Improvement Rates at Indiana's High-Improving, 
Average, and Low-Improving Schools: Reading 
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Figure 13: Demographics of Indiana's High-Performing, Average, 
and Low-Performing Schools: Reading 
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Figure 14: Demographics of Indiana's High-Improving, Average, 
and Low-Improving Schools: Reading 
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Demographics of Schools Differing in Performance and 
Improvement. Looking at the demographics of high- 
performing, average, and low-performing schools, we 
again see that schools in the bottom quartile have the 
highest percentages of low-income and minority students 
(see Figure 13). Unlike in Maryland, however, Indiana's 
high-improving, average, and low-improving schools have 
similar demographics (see Figure 14). In other words, the 
data indicate that Indiana schools that serve high percent- 
ages of low-income and minority students are more likely 
than other schools to be low performing, but they are no 
more likely to be making high gains. 

Where Schools at Different Performance Levels Started 
and Ended Up. Figure 15 shows the 2004-08 proficiency 
rates of schools that started out high performing, aver- 
age, and low performing. On average, reading proficiency 



rates at schools that started out high performing declined 
slightly. Average-performing schools stagnated, neither 
losing ground nor improving over this period. Schools that 
started out low performing did make small gains, but with 
average improvement rates of just over half a percentage 
point per year, there is little evidence that these schools are 
catching up with their higher performing counterparts. 

What's more, a closer look at trends in proficiency rates 
of initially low-performing schools shows that while about 
38 percent of these (141 schools) become among the high- 
est improving in the state, nearly a quarter (88 schools) are 
lowest improving (see Figure 16). 

In fact, because reading performance at the 25th per- 
centile school is actually declining by a percentage point 
per year, we know that some of the average-improving low 
performers are also losing ground. Because any low-per- 
forming school that is declining is, by definition, stuck, the 
number of stuck schools in Indiana actually is greater than 
the 88 schools in Figure 16. Indeed, 141 schools started 
out low performing and proceeded to lose ground — just 
under 40 percent of the state's low performers (see Figure 17 
on page 9). 

Figure 18 shows the 2004-08 performance of these 141 
stuck schools and other initial low performers. Although, 
on average, proficiency rates at high-improving schools 
increased by about ten percentage points over this period, 
stuck schools declined by five percentage points. And in 
some schools, declines have been far more dramatic: At an 
elementary school in Richmond that serves more than 350 
students, for example, the percentage of those who scored 
proficient or above in reading fell from 64 percent in 2004 
to only 50 percent in 2008. 



Figure 15: 2004-08 Reading Performance of Indiana's 
High-Performing, Average, and Low-Performing Schools 



100 







4 


► 


A- 


# 


▲ 






IB 


80 










▼ 




▼ 


| 








— m — 


m 


• 






1 


60 


Baseline State Average 












"p 

cfc 


40 


■ 


— 




■ — 






M 


a? 
















20 




















0 




















2004 


2005 


2006 


2007 


2008 








^ High Performing 


# Average Performing 


| Low Performing 







Figure 16: Number of Low-Performing Schools in Indiana That Are 
High Improving, Average, and Low Improving: Reading 
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Figure 17: Number of Low-Performing Schools in Indiana That Are 
High Performing, Average, and Low Improving: Reading 

(Low-Improving schools include all schools with declining 
performance) 




Identifying Chronically Low-Performing Schools. Because 
so many schools in Indiana have lost ground from 2004 
to 2008, low-performing schools that are not declining 
will not show up in our data as stuck schools, even if they 
are making only minimal gains. Examples of such "non- 
stock" low-performers include a middle school where 37 
percent of students were proficient in reading in 2004 and 
40 percent reached this target by 2008. Another example 
is an elementary school where reading proficiency rates 
"improved" from 40 percent to 41 percent. Considering 
that in our baseline (2004-06), the fifth-percentile school 
(the school whose proficiency rates were lower than those 
of 95 percent of schools in Indiana) had a reading-profi- 
ciency rate of 50 percent, results at both of these schools 
are utterly dismal. 

All of this is to say that in Indiana, as well as in any 
state where all low-improving schools are losing ground, it 
is particularly important to identify schools where perfor- 
mance remains chronically low. 

To identify chronically low-performing schools in Indi- 
ana, we used the same approach as in Maryland. Compar- 
ing 2006, 2007, and 2008 reading-proficiency rates of each 
school with the performance of the fifth-percentile school 
in the baseline reveals that 31 schools consistently scored 
below this benchmark. Seventeen of these schools are also 
stuck (they lost ground from 2004 to 2008); the remaining 
14 (including the two aforementioned schools) are making 
small but insufficient gains — in some cases less than one 
percentage point each year (see Figure 19). 



Figure 18: 2004-08 Reading Proficiency Rates of Indiana's 
Low-Performing Schools Showing High, Average, and Low 
Improvement 
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Figure 19: Indiana's Chronically Low-Performing Schools: Reading 




Number of Schools 


Exited bottom 5 percent 


31 


Exited bottom 5 percent but declined again 


11 


In bottom 5 percent for all three years 


31 



Tallying Stuck and Chronically Low-Performing Schools. 

Based on 2004-08 ISTEP+ reading scores, 155 elementary 
and middle schools in Indiana are stuck, chronically low 
performing, or both (see Figure 20 on page 10). ISTEP+ 
math results, which reveal 147 stuck or chronically low- 
performing schools, appear in Appendix II. A closer look 
at these schools shows that 74 are stuck or chronically 
low performing in both reading and math, 81 in reading 
only, and 73 in math only. In total, 228 schools (about 
15 percent of elementary and middle schools in Indiana) 
started out low performing in reading, math, or both and 
have proceeded to decline or make minimal gains over the 
subsequent five years (see Figure 21). 

Trends in Indiana's Data. On average, reading and math 
proficiency rates in Indiana's elementary and middle 
schools have remained nearly flat from 2004 to 2008 at 
high-performing, average, and low-performing schools. 
More than one-third of schools that started out in the bot- 
tom quartile of performance in the baseline in each subject 
proceeded to lose ground over this time. A few more 
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Figure 20: Overlap Between Indiana Schools That Are Stuck and 
Chronically Low Performing in Reading 



Figure 21: Overlap Between Indiana Schools Identified as Stuck or 
Chronically Low Performing in Reading and Math 





low-performers managed not to decline but continued to 
consistently perform below 95 percent of the schools in 
the state — a clearly unacceptable level. 

Yet even in Indiana, where the overall picture is quite 
dim, some schools demonstrate that improvement is pos- 
sible. At an elementary school in Seymour, for example, 
reading-proficiency rates rose from 48 percent in 2004 to 
72 percent in 2008. And at an elementary school in Gary, 
35 percent of students passed the math ISTEP+ in 2004, 
but 61 percent passed in 2008. Indiana could benefit from 
finding these schools, understanding what they are doing 
to raise proficiency rates, and taking meaningful improve- 
ment strategies to scale. 

CONCLUSION AND IMPLICATIONS 

So what do these data tell us about recent patterns 
in school improvement? Tracking math and reading- 
assessment results over time, we see that some schools 
that started out low-performing are making substantial 
progress. Their gains aren't always consistent from year to 
year, but proficiency rates at these schools are clearly and 
substantially improving. Most of these schools still aren't 
performing as well as they should, but students and educa- 
tors would benefit if state and district leaders figured out 
what these schools are doing right and scaled up practices 
that are likely to lead to meaningful gains in learning. 

Each of the ten states we looked at, however, also has 
another group of schools — ones that started out low 
performing and have made little to no improvement or 
even lost ground. What's more, some schools persistently 
produced worse results than 95 percent of schools in their 
states, even as they managed to make some gains. In an 



environment where funds and capacity are limited at best, 
educators and policymakers will need to establish clear 
priorities. Together with AYP data and other indicators, 
this analysis can help states focus resources where they 
are most needed — on ensuring that students currently 
stranded in low-performing schools that can't or won't 
improve on their own get the support they need to catch 
up to their peers. 

Finally, as data from Maryland and Indiana demon- 
strate, performance and improvement patterns can differ 
dramatically from state to state: Maryland shows a pattern 
of overall gains; Indiana, general stagnation. We selected 
these states precisely because they present such different 
trends — ones we have observed in other states. Again, 
Maryland and Indiana are merely examples, and our inten- 
tion is not to show that one is doing better than the other. 
Rather, it is to highlight these trends and, in the process, 
offer ways to help educators and policymakers explore data 
in their own states. 

Of course, overall test scores can only tell us so much. 
The next paper in this series will examine the performance 
of student subgroups. And following that, we will take a 
closer look at where stuck schools operate to see whether 
improvement patterns differ from district to district. 

Likewise, it's important to consider these data within 
the broader policy context. Many educators and policy- 
makers agree that the next generation of accountability 
systems must measure absolute performance and prog- 
ress alike. The diverging trends we found among schools 
that start out low performing support this notion. These 
schools are not all the same: An accountability system 
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should distinguish between schools that start out low 
performing and make gains and those that year after year 
show little capacity to improve. 

That said, the comparative nature of this analysis limits 
its use in developing a new accountability system. If 
anything, the drastically different improvement patterns 
in Maryland and Indiana offer a cautionary tale to those 
who insist that performance targets should be based solely 
on comparative data. When, as in Indiana, it takes gains 
of one percentage point per year to be considered a top 
improver, pegging expectations to the current rate of prog- 
ress isn't nearly enough. 

Lessons from this analysis can help inform the account- 
ability conversation, and the final paper in this series 
will explore these in greater depth. While there is much 
more to learn about patterns in school improvement, we 
hope the approach outlined here offers states and districts 
another way of looking at their own data to find schools 
whose students need help not tomorrow, not in a year or 
two, but now. 



An accountability system 
should distinguish between 
schools that start out low 
performing and make gains 
and those that year after year 
show little capacity to improve. 



NOTES 

(Some links may have expired. Some links that appear on multiple lines may not be reachable directly from this document. 
It may be necessary to copy and paste the entire link into your browser.) 



1 Until the 2008-09 school year, Indiana administered the Indiana 
Statewide Testing for Educational Progress-Plus (ISTEP+) assess- 
ments for grades 3-8 in the fall of each school year. In 2008-09, 
the assessment was administered twice — once in the fall and again 
in the spring. Although the fall 2008 results are comparable with 
those of prior years, the spring 2009 assessment has changed 
substantially and no longer is comparable. Our analysis includes 
assessment results from the fall 2004 to fall 2008 assessments. We 
refer to these data as 2004-08 to be consistent with the state and 
to reflect the fact that fall assessments measure learning from the 
prior academic year, rather than the current year. 

2 We calculated baseline proficiency rates by averaging the first three 
years of each school's test results (2004-06 in Indiana and 2005-07 
in Maryland). To calculate average annual improvement, we used 
the formula for the slope of the best fit-line running through each 
school's proficiency rates over the five years analyzed. 

3 "Mapping State Proficiency Standards onto NAEP Scales: 2005- 
2007" National Center for Education Statistics, 2009. 

4 Two things to note about the Maryland data used in this analysis. 
In 2008, Maryland eliminated a number of norm-referenced test 
items that had not counted toward a student's score on the Mary- 
land State Assessments in reading and math. It also replaced sev- 
eral additional items that had counted with field-tested questions 



developed by the state. Following a review of the 2008 test, Mary- 
land's Psychometric Advisory Council concluded that the slightly 
shorter 2008 assessments in both subjects were comparable in dif- 
ficulty with prior administrations of the MSA. But it acknowledged 
the possibility that the "reduced testing burden on the students" 
may have contributed to the observed gains in test scores that year 
(National Psychometric Council. August 1, 2008. Memorandum 
to the Maryland State Board of Education Re: 2008 MSA Linking. 
Available at www.msde.maryland.gov/NR/rdonlyres/3253ClDD- 
CA2E-4E64-A066-D6F36EBADF9B/179S)7/2008MSAresultsbriefi%20 
ngpaperAug08F.pdf. Maryland, along with organizations such as the 
Center on Education Policy, compares 2008 results with those of 
prior years. However, because of the changes mentioned above, we 
urge readers to use caution when interpreting the data presented 
in this section. Furthermore, note that this analysis relies on 
proficiency-rate data used to make AYP determinations in the state. 
Changes to how these proficiency rates are calculated may affect 
improvement-rate estimates. 

5 While alternative and special education schools may require state 
or district intervention, it may not be appropriate to compare 
them with regular schools. We have chosen to exclude them from 
this analysis and will discuss this decision in more detail in a 
subsequent paper. 
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Appendix I 



"Stuck Schools'' presents a comparative analysis of perfor- 
mance data and improvement trajectories of elementary 
and middle schools in a state. The analysis is based on 
school-level state-assessment data for five consecutive 
years (2004-08 in Indiana and 2005-09 in Maryland). 
Aggregate schoolwide proficiency rates in reading and 
math are used to (1) rank and assign schools to a base- 
line performance category and (2) assess change in 
performance over time. Content standards, performance 
standards, and assessments must be comparable across 
the time period to conduct the analysis. 

Schools are identified as being stuck in reading, math, 
or both based on low baseline performance and low 
improvement over time. In addition, chronically low per- 
forming schools are also identified and cross-referenced 
with "stuck schools" to determine the overlap between 
schools in each of these categories. 

In "Stuck Schools," publicly available data in reading 
and mathematics from Maryland and Indiana are used to 



walk readers through the analytical framework. Each state 
provides the data for core variables in slightly different ways. 

SCOPE OF THE ANALYSIS 
I. Variables and Data Sources 

The variables needed to conduct the analysis are: 

• School characteristics 

- Percentage minority students (African American, 
Latino, or Native American) 

- Percentage low-income students (eligible for free 
or reduced-price lunch) 

• Aggregate schoolwide proficiency rate (percentage 
of all students in grades 3-8 scoring proficient or 
above on the state assessment) in reading and math, 
respectively, for five consecutive years (2005-09 in 
Maryland and 2004-08 in Indiana) 

Table A-l lists and describes the data sources used in 
the Maryland analysis. Table A-2 lists and describes the 



Table A-1: Maryland Data Sources* 



Data Sought 


Data Source 


Data Available 


Data Source URL 


2005-09 Proficiency Rates 


2005-09 Adequate Yearly Prog- 
ress (AYP) files 


Total number and percentage 
of students scoring proficient 
or above in reading and math, 
by subgroup, in each Maryland 
public school. 


http://mdreportcard.org/down- 
loadindex.aspx (2009) and http:// 
mdreportcard.org/downloadin- 
dexprevious.aspx?k=99AAAA 
(2005-2008) 


Percentage Low-Income and 
Percentage Minority 


2009 Adequate Yearly Progress 
(AYP) file 


Total number of students tested 
and testing participation rate, 
by subgroup and by school, for 
each Maryland public school. 


http://mdreportcard.org/down- 

loadindex.aspx 


Alternative/ Special Education 
School Identifier 


2009 Students Receiving Special 
Services file 


Percentage of students receiv- 
ing special education services in 
each Maryland public school. 


http://mdreportcard.org/down- 

loadindex.aspx 


High School Grades Identifier 


2009 Enrollment Data file 


Total enrollment by grade (but 
not by subgroup) in each 
Maryland public school. 


http://mdreportcard.org/down- 

loadindex.aspx 



* 2005-2008 Adaquate Yearly Progress files were downloaded in July 2009. All other data files were downloaded in October 2009. 
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Table A-2: Indiana Data Sources* 



Data Sought 


Data Source 


Data Available 


Data Source URL 


2004-08 Proficiency Rates 


2004-08 Public School Disag- 
gregated Files for Grades 3-8 


Total number of students tested 
and number of students scoring 
proficient or above in English/ 
Language Arts (reading) and 
math, by subgroup and by grade 
in each Indiana public school. 


www.doe.in.gov/istep/2008/welcome.html 

www.doe.in.gov/istep/2007/welcome.html 

www.doe.in.gov/istep/2006/welcome.html 

www.doe.in.gov/istep/2005/welcome.html 


Percentage Low-Income 


2008-09 Free Lunch Counts 
by School 


Number of students qualifying 
forfree lunch and reduced- 
price lunch, by school, for each 
Indiana public school. 


http://mustang.doe.state.in.us/SAS/sas2. 

cfm?type=s&tab=sl&already= 


Percentage Minority 


2008-09 Enrollment by Grade 


Total number of students and 
number of students by race 
enrolled in each Indiana public 
school. 


http://mustang.doe.state.in.us/SAS/sas2. 

cfm?type=s&tab=peschl&already= 


Alternative/Special Educa- 
tion School Identifier 


General School Variables 


Contains a School Type vari- 
able that identifies alternative 
(Stype=9), special education 
(Stype=7), and other special- 
ized types of schools (e.g. adult 
education programs) 


http://mustang.doe.state.in.us/SAS/sas2. 

cfm?type=s&tab=schls&already= 



*2004-08 Public School Disaggregated Files for Grades 3-8 and the General School Variables file downloaded in July 2009. 2008-09 Free Lunch Counts and Enrollment by Grade downloaded in October 2009. 



data sources used in the Indiana analysis. 

The various available data sets in each of the states were 
merged together to perform the analysis. Prior to ranking 
schools on performance and improvement, schools that 
did not meet the analysis criteria were eliminated from the 
dataset. The following types of schools were excluded from 
the analysis: 

1. Schools missing proficiency- rate data for one or more 

years. 

2. Schools missing key directory information. 

a. Maryland: schools missing grade-by-grade enroll- 
ment data 

b. Indiana: schools missing school-type data 

3. Alternative schools and special education schools 

a. Maryland: Schools serving more than 50 percent 
of students receiving special education services 
were excluded, as were education centers, hospital 
facilities, and schools that included "alternative" 



in their school name. 

b. Indiana: A school type variable was provided by 
the state that classified schools as special educa- 
tion (Stype=7) or alternative (Stype=9). 

4. Schools or data that included high school assessment 

results 

a. Maryland: Because Maryland's AYP files did not 
provide grade-by-grade proficiency rates, schools 
with enrollment in grade 9 or higher were 
excluded. 

b. Indiana: High school-level test results were 
excluded from the analysis. Schools that served 
some combination of grades 3-8 and high school 
grades remained in the analysis. 

See Tables A-3 and A-4 for counts of schools that were 
included in the analysis or eliminated based on the above 
criteria in Maryland and Indiana, respectively. 
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Table A-3: Number of schools included in and eliminated from the analysis, by school type: Maryland 



School Type 


Reading Analysis 


Math Analysis 


Total number of schools with any assessment data for 2005-09 


1,458 


1,458 


Total number of schools with five years of data 


1,281 


1,283 


Eliminated from analysis Schools missing grade-by- grade enrollment data 


0 


0 


Alternative/special education Schools 


23 


23 


High Schools (no elementary /middle school grades) 


184 


186 


Elementary /middle schools with high school grades 


8 


8 


Total schools included in analysis 


1,066 


1,066 


Table A-4: Number of schools included in and eliminated from the analysis, by school type: Indiana 


School Type 


Reading Analysis 


Math Analysis 


Total number of schools with any assessment data for 2004-08 


1,669 


1,669 


Total number of schools with five years of data 


1,488 


1,488 


Eliminated from analysis Schools missing school-type data 


5 


5 


Alternative/ special education schools 


6 


6 


Total schools included in analysis 


1,477 


1,477 



In addition, the percentage of low-income and minority 
students was calculated for each school. Minority students 
are defined as students who are African American, Latino, 
or Native American. Low-income students are defined as 
those eligible for free or reduced-price lunch. For Indiana, 
the demographic characteristics of each school were calcu- 
lated based on enrollment data and free or reduced-price 
lunch count data. For Maryland, these characteristics were 
estimated for each school based on the number of students 
tested and the participation rate, as follows: 

• Percent Minority= [ [Number of African American 

Students Tested/ (Participation Rate,,. , /100) 

+ Number of Latino Students Tested/(Participa- 
tion Rate Latino /100) + Number of Native American 
Students Tested/ (Participation Rate.. , , . / 100) | 

/ [Total Students Tested/ (Participation Rate A|1 students 

/ 100 )]] 

• Percent Low-Income=[ [Number of students eligible 
for free or reduced- price lunch tested/(Participation 
Rate, . . ,. ....../ 100)1 / [Total Students Tested/ 

(Participation Rate,,, /100)] 

Where the participation rate was not reported, we 
assumed a participation rate of 100 percent. 



II. ANALYSIS OF PERFORMANCE 
AND IMPROVEMENT OVER TIME 

The following steps and calculations were used to look at 
school performance over time and to identify stuck and/or 
chronically low-performing schools in each state. Steps 1-9 
were performed separately for reading and math assess- 
ment results. 

A. Calculating Baseline Performance 
and Improvement Rates 

1 . Calculate the aggregate schoolwide proficiency rate 
for each school: 

Aggregate school proficiency rate (SubjMA)(YearN) =[Sum 
of all students in grades 3-8 that scored proficient or 
above (SubjectA)(YearN) /[Total number of students tested in 
grades 3-8j SubjectA jj YearN j] 

2. Calculate a baseline performance for each school by 
averaging that school's proficiency rates across the 
first three years of the analysis time period (2004-06 
for Indiana and 2005-07 for Maryland) 
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3. Calculate each school's improvement rate (over 
2004-08 for Indiana and 2005-09 for Maryland) 
using the following formula for the slope of the best- 
fit line: 

Improvement rate = n(Zxy) - (Zx)( Zy) 
n(Zx 2 ) - Ex ) 2 

Where 

x= School year (2004-08 for Indiana; 2005-09 for 
Maryland) 
y= proficiency rate 

And where 

Zxy = sum of products = + x 2 y 2 + . . . + x n y n 

Zx = sum of x-values = x + x 2 + . . . + x n 

Zy = sum of y-values = y + y 2 + . . . + y n 

Zx 2 = sum of squares of x-values = x 2 2 + x 2 2 + . . . + x n 2 

B. Categorizing Schools Based on Performance 
and Improvement; Identifying Stuck Schools 

4. Rank schools by baseline performance and divide 
into quartiles. Classify schools whose baseline per- 
formance was in the top 25 percent of all the state's 
schools as "High Performing," those in the middle 
50 percent as "Average Performing," and those in the 
bottom 25 percent as "Low Performing." 

5. Rank schools by improvement rate and divide into 
quartiles. Classify schools whose improvement rate 
was in the top 25 percent of all the state's schools as 
"High Improving," those in the middle 50 percent 
as "Average Improving," and those in the bottom 25 
percent as "Low Improving." All schools that lost 
ground during the time period analyzed were consid- 
ered low improving. 

6. Calculate descriptive statistics, including percentage 
minority, percentage low-income, and average per- 



formance by year for schools at each level of perfor- 
mance and improvement. 

7. Identify which of the low-performing schools are 
high, average, or low improving. Schools that are 
low performing and low improving are classified as 
"stuck." 

8. Calculate average proficiency rates by year for low- 
performing schools showing high, average, and low 
improvement. 

C. Identifying Chronically Low-Performing 
Schools 

9. Identify schools whose performance in each of the 
last three years analyzed (2006, 2007, and 2008 for 
Indiana and 2007, 2008, and 2009 for Maryland) 
was below the proficiency rate of the fifth-percentile 
school in the baseline. Schools whose performance 
was below this benchmark in all three years are clas- 
sified as chronically low performing. 

D. Determine Overlap Between Stuck 
and Chronically Low-Performing Schools 

10. Cross-reference list of schools identified as stuck and 
those identified as chronically low performing in 
each subject to (a) distinguish between schools that 
are both stuck and chronically low performing and 
those that fall into only one of these categories, and 
(b) to calculate the total number of schools identi- 
fied in a particular subject. 

11 . Cross-reference list of schools identified as stuck or 
chronically low performing in reading and math to 
(a) distinguish between schools that are struggling 
in both subjects and those struggling in only one, 
and (b) calculate the total number of schools in the 
state identified as stuck or chronically low perform- 
ing in at least one subject area. 
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Appendix II 



T his appendix presents the results of our analysis of math performance over time at elementary and middle schools 
in Maryland and Indiana, respectively. Figures B-l through B-9 parallel Figures 1-9 for reading in the main report, 
while Figures B-10 through B-19 parallel Figures 11-20. 



I. MARYLAND MATH RESULTS 



Figure B-1: Baseline (2005-07) Math Proficiency Rates of Figure B-3: Demographics of Maryland's High-Performing, 

Maryland's High-Performing, Average, and Low-Performing Average, and Low-Performing Schools: Math 

Schools 



High 

Performing 



Average 

Performing 



Low 

Performing 



91% of students 
pass 



76% of students 
pass 



51% of students 
pass 



Maximum 

99% 



75th percentile 
86 % 



25th percentile 
65% 



Minimum 

6 % 




Figure B-2: 2005-09 Improvement Rates at Maryland's High- Figure B-4: Demographics of Maryland's High-Improving, Average, 

Improving, Average, and Low-Improving Schools: Math and Low-Improving Schools: Math 



High 

Improving 



Average 

Improving 



Low 

Improving 



6.4 percentage 
points/year 



2.4 percentage 
points/year 



0.2 percentage 
points/year 



Maximum 

18.2 percentage 
points/year 

75th percentile 

4.2 percentage 
points/year 



25th percentile 
1.1 percentage 
points/year 

Minimum 
-4.7 percentage 
points/year 
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Figure B-8: Maryland's Chronically Low-Performing Schools: Math 



Figure B-5: 2005-09 Math Performance of Maryland's 
High-Performing, Average, and Low-Performing Schools 
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Number of Schools 



Exited bottom 5 percent 


24 


Exited bottom 5 percent but declined again 


1 


In bottom 5 percent for all three years 


17 
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Figure B-6: Number of Low-Performing Schools That Are High-, 
Average, and Low Improving: Math 




Figure B-9: Overlap Between Schools That Are Stuck and 
Chronically Low Performing in Math in Maryland 




Stuck and Chronically 
Low Performing 



Figure B-7: 2005-09 Math Proficiency Rates of Maryland's 
Low-Performing Schools Showing High, Average, and Low 
Improvement 
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II. INDIANA MATH RESULTS 



Figure B-10: Baseline (2004-06) Math Proficiency Rates of Figure B-12: Demographics of Indiana's High-Performing, 

Indiana's High-Performing, Average, and Low-Performing Average, and Low-Performing Schools: Math 

Schools 



High 

Performing 



Average 

Performing 



Low 

Performing 



87% of students 
pass 



76% of students 
pass 



59% of students 
pass 



Maximum 

97% 



75th percentile 
82% 



25th percentile 
69% 



Minimum 

26% 



100 



80 

71 




High Performing Average Performing Low Performing 



■ Minority Low Income 



Figure B-11: 2004-08 Improvement Rates at Indiana's Figure B-13: Demographics of Indiana's High-Improving, Average, 

High-Improving, Average, and Low-Improving Schools: Math and Low-Improving Schools: Math 



High 

Improving 



Average 

Improving 



Low 

Improving 



2.9 percentage 
points/year 



0.2 percentage 
points/year 



-2.2 percentage 
points/year 



Maximum 

1 1 .9 percentage 

points/year 

75th percentile 
1.5 percentage 
points/year 



25th percentile 
-1.0 percentage 
points/year 

Minimum 
-7.6 percentage 
points/year 



100 

80 

£3 



<£ 40 — 

27 26 

20 18 1 

■ ■ ■ 

High Improving Average Improving Low Improving 

■ Minority Low Income 
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Figure B-14: 2005-09 Math Performance of Indiana's 
High-Performing, Average, and Low-Performing Schools 
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Figure B-17: 2005-09 Math Proficiency Rates of Indiana's 
Low-Performing Schools Showing High, Average, and Low 
Improvement 
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Figure B-15: Number of Low-Performing Schools in Indiana That 
Are High-Improving, Average, and Low Improving: Math 

(Low-Improving schools include only schools in the bottom quartile 
of improvement) 




Figure B-18: Indiana's Chronically Low-Performing Schools: Math 



Number of Schools 



Exited bottom 5 percent 


34 


Exited bottom 5 percent, but declined again 


6 


In bottom 5 percent for all three years 


32 



Figure B-16: Number of Low-Performing Schools in Indiana That 
Are High Improving, Average, and Low Improving: Math 

(Low-Improving schools include all schools with declining 
performance) 




Average 
Improving, 
82 ( 22 %) 



Figure B-19: Overlap between Indiana Schools that are Stuck and 
Chronically Low Performing in Math 




Stuck and Chronically 
Low Performing 
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